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ABSTRACT 

Nonlinear programming methods are used to solve saddle-point prob- 
lems subject to inequality constraints on the variables; in particular, the 
type of saddle-point problem arising in pursuit-evasion differential games 
is considered, The methods investigated fall into two groups: solution 
of the nonlinear simultaneous equations obtained from the Kuhn-Tucker 
conditions, and solution of a sequence of constrained optimization prob- 
lems by the gradient projection algorithm. These methods are applicable 
to any real-valued function f(x,y) which is convex in x, concave in y, and 
has continuous and bounded second partial derivatives. Several examples 


are given which illustrate the characteristics of the numerical procedures. 
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I. INTRODUCTION 
The original development of differential game theory is credited to 
Isaacs for his work during the 1950's [8]. His book [9], published in 
1965, provides a comprehensive summary of this theory. Issacs' develop- 
ment closely resembles the dynamic programming approach to optimal 
control theory popularized by Bellman [2]. Many subsequent authors hawe: 
studied differential games; among the significant contributors are Ho, 
Bryson, and Baron [6], and Berkowitz [3]. These and many recent papers, 
use the calculus of variations as an analytical tool. 
A statement of the differential game problem is as follows: A dyna- 
mic system is described by the state equation 
Xuseil(tg ti (Magid tes Drees > anal aa (1) 
where xis a vectorwhich describes the position,,or state, of theagame, 
u is a control,vector selected by player 1 andwwwis.a conimelwcarenmssolee— 
ted by player 2. The vectors u.and v are members of the sets Ueandey 
respectively, denoted by weU and v ¢VeewhereUsand Veareutoske speci- 
fied later. The performance criterion is a functional assumed to be of the 
form 


P= helt), te) + fF Get, ule), vO, Hat, (2) 


0 


which player 1 wishes to maximize, and player 2 wishes to minimize. A 
saddle point of J is sought (if it exists), i.e., au*e U and v*eV are 


sought such that 


Jtu,v*) sJ(u*, v*) <= J(u*, vy) (3) 

In the terminology of game theory, J is the payoff, u and v are strategies 
of the two opposing players, and u* and .v* are optimal pure strategies. 
A strategy is a decision rule that specifies a control to be applied by a 
player, based on all information available to him, in any given situation. 
Generally, the literature has considered a strategy to be a feedback con- 
trol law, that is, u and v are specified in terms of the states x; however, 
Willman [20] has used the concept of an open-loop strategy. In terms of 
the previous definition, this means that the information available to each 
player is restricted to the initial state and time, and the terminal time. 
Were tharur-and v* are nimimad<strategies in te sense tha, player 1 
maximizes his minimum gain while player 2 minimizes his maximum loss. 

The concept of fixed terminal time ignores the fact that in more 
general games the game may not terminate at all. Berkowitz [4]requires 
a differential game to terminate whenever t and x(t) are such that 
(t,x(t)) is a point of a previously specified set in (t,x) space. This is 
the "terminal surface" discussed by Isaacs [9]. Berkowitz [4]lalso 
restricts the term "differential game" to include only two-person, zero-~ 
sum games (the type described above), although others [19] have not 
been so restrictive and have extended the theory to N-player, non~zero- 
sum games. Zero-sum refers to the fact that there is a single performance 
criterion which one player tries to maximize while the other tries to min- 
imize. Thus, the maximizing player's gain is the minimizing player's loss 
and the sum of this gain and loss is zero. In a non-zero-sum game, the 


objectives of the two players are not directly opposed -- each may have 
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an individual performance measure to minimize, and the sum of the two 
players' criteria is not necessarily zero. It may also be that there are 
more than two players, each controlling an input to a single system, and 
each trying to minimize his individual performance criterion. Such prob- 
lems are not considered in this thesis. 

Although variational techniques seem to be an appropriate method 
for solving differential games, a distinct drawback is that inequality 
constraints on the value of the state and control variables at any time 
are difficult to include in the analysis [4]. Thus, more tractable methods 
are sought. Since the problem is basically an optimization problem sub- 
ject to inequality constraints it is natural to consider mathematical pro- 
gramming techniques (nonlinear programming in particular since the 
performance measure will generally be nonlinear). As far as this author 
can determine, only one other paper [10] has considered this approach. 

The nonlinear programming problem was first stated mathematically 
by Kuhn and Tucker [14]. The problem is to find an x° that maximizes a 
function F(x) subject to inequality constraints of the form X{x)20* and x 
20. Kuhn and Tucker found necessary conditions for x° to be a solution 
to this problem. If F(x) is a concave function in the region where 
A(x) 20 and x 20, the conditions are also sufficient. Several algorithms 
to solve this problem have been proposed although this thesis will use 


only Rosen's gradient projection algorithm (15]. 


A(x)20 means that each component of the vector A(x) is non- 
© 


Negeri 


The use of nonlinear programming methods to solve optimal control 
problems has been discussed by Rosen [16], [17], and by Kirk [12]. 
This thesis will discuss the connection between nonlinear programming 
and saddle-point problems such as arise in pursuit-evasion differential 


games, and several examples which illustrate these ideas will be solved. 
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II, BASIC PRINCIPLES 

Several results from Game Theory will be required and are stated 
here. Amore complete discussion of these theorems and their proofs can 
Dpemeatumeliei mie ).), |, x: 
Theorem 1. General Min-Max Theorem. Let f(x, y) be a real-valued 
function ofvtwo vectors x and y which lie in X and Y, respectively, where 
both X and Y are closed, bounded, convex sets.” If f is continuous, 
convex in x for each y, and concave in y for each x,* then 


in ax f(x, = max 
mi m vee (x,y) =m 


xe X eT elegy 


B7cey, 
A sufficient condition for a saddle point follows from the next 

Eee Lem. 

iheorem.2. A necessary and sufficient condition for 


min max f(x,y) = max 


xe X ye Y ye Y ane c ay 


is that there exist a pair x*¢X, y*e Y such that for all y in Y and all 


xin Xx 


2X is aconvex set if the straight line joining any two points 
my A lies entirely within X, THatis, ii x > x ~ eX then 


x* =@x* + (1-9@x? ¢€X forall 0<@<1 
° f(x) is a convex function if 
(l-6)f(x° )+6fx*) = £f(fl-9)x° + 6x? ) (i) 


for 0<6<1andforallx° and x* . The negative of a convex 
function is a concave function, i.e., f(x) is a concave function if the 
Sign of the inequality in (i) is reversed. 


1 


f(x*,y) sfi(x*,y*) <f(x,y*) ; (4) 
Thus, a sufficient condition for the existence of a saddle point (in 
the sense of (4) above) is that f(x, y) be convex in x for all y and concave 
Iie ole ee 
Considering each side of the double inequality in (4) separately, 
the problem can be viewed as two optimization problems. Looking at the 
right-hand inequality first, and specifying the set Y as Y = {y| di (2 Un 


i=1,...,4}, the problem 


ax f(x*, 
Nee ee 


where y is an m-=vector, 1s a Statement of a Maximum Preplem from mone 
linear programming. If f(x,y) has continuous first partial dérivatives, 
the Kuhn-Tucker necessary conditions can be written.* For y* tobea 
solution to the above problem it is necessary that y* and themsveciomon 


Lagrange multipliers Ky Satisfy the following conditions 


£ 
far) +B Uh yy ee = a : 
r ve)” a =. (6) 
rs, ly*) @e Oo (7) 


* The original problem considered by Kuhn and Tucker also included 
a HNonmecCadving@ivecetomaint Onmune variables; ~ye0 , bul general Usage 
has associated the term Kuhn-Tucker conditions with the necessary con- 
ditions for this problem as well as the one they originally considered. 


where a f{(x*, y*) denotes the gradient of f(x*, y) with respect to y 


evaltiatcatalL a, lec 


af (x*, y) 


ey, 


Sf (x*, y) 


0 ee ee ee a ee 


ty Oy, tap Y — y* 
and that the constraint qualification [14]be satisfied. The constraint 
qualification rules out pathological behavior on the boundary of the con- 
straint set. In this thesis, only linear constraints will be considered, 
hence the constraint qualification is always satisfied. It has been shown 
[14] that if f{(x*, y) is a concave function of y, then the conditions (5)- 
(8) are also sufficient. 
Similarly, for the problem, 
aes {(x, y*) 


subject to r,, &) > 0 ; i=ve2. Fak 


Falbiet ad Convex differentiable function Of x, an N-Vvecten, Inemieccrcam, 


and sufficient conditions are, 


k 
a a De ee (9) 
X (x*)i p = 0 (10) 

mm ».4 
A (*) 2 0 (11) 
ye 2 il) (12) 


where L. is a vector of Lagrange multipliers. Thus, for (x*, y*) to be a 


solution of the constrained saddle-point problem, equations (5), (6), (9), 


= ~ 
WAS 


and (10) and inequalities (7), (8), (11), and (12) must be satisfied. 
Although the Kuhn=-Tucker conditions require only that the function be 
real-valued, concave-convex, and differentiable to ensure that a saddle 
point exists, the methods examined in this thesis will also require the 


function to have continuous and bounded second partial derivatives. 
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III, NUMERICAL METHODS 
The methods investigated for solving the constrained saddle-point 
problem may be divided into two groups; 1) solution of the nonlinear 
simultaneous equations given by the Kuhn-Tucker conditions, and 2) 
direct solution of a maximization problem and a minimization problem. 
Methods in the first group have the disadvantage that although the original 
problem is of dimension m+n, the Langrange multipliers must also be 


treated as variables and the dimensionality increases to m+n+k+l. 


A, SOLUTION OF NONLINEAR SIMULTANEOUS EQUATIONS 
1, Newton-Raphson Method 
Jacob and Polak [10] suggest a generalized Newton-Raphson 
method for solving the system of four(vector) equations (5), (6), (9), and 
(10) subject to the four(vector) inequalities (7), (8), (11), and (12). Con- 
vergence of this method was found to be somewhat sensitive to the 
starting point, although convergence was quite rapid if appropriate start- 
ing points were chosen. 
2. Brown's Algorithm® 
Brown [5] has proposed an algorithm to solve nonlinear simul- 
taneous equations which requires fewer multiplications than the Newton- 
Raphson method. Itis quite similar to the Gauss-Seidel process for 


nonlinear systems of equations. Brown has observed that the stability 


© Subroutine NLNSYS at NPS Computer Facility. 


le 


and convergence df the algorithm do not seem to be dependent on the 
exact evaluation of the partials and first differences may be used to 
approximate the partials, This method was also found to be sensitive to 
the starting point. 


3. Gradient Projection Determination of the Roots of the Equations 


given by the Necessary Conditions® 





This method suggested itself due to the nonnegativity con- 
straints on the multipliers in the necessary conditions. There are several 


ways of formulating this problem; the one chosen was to minimize 


as T 2 dina ce 

pe OA @o uP + wo Hw (13) 
subject to 
kk 
Pyne, yh ee Ae = 0 
= Se eS 
f{ = eal 0) 
4 se 

7 £(x,y) me ” yi WY 0 
A.) 2 0 (14) 

ae) 

Ay) = 0 

Le = 0 

: 0 

By eu 


Although the gradient projection algorithm can be altered to include 
equality constraints [15], they may also be written as two inequalities, 


i.e. 


=the gradient projection algorithm programmed by D. E. Kirk was 
used in this and the following two applications. 


LO 


Om. = Ler) +2, es Ag A, &) J 20, 


(15) 


O-< [7 fey +2 uw, 7 Aw Iso, 


The Mintinumevalue oO: ft will beszerovaneethc Valuesson amy. bs ee 


yielding this minimum will satisfy the Kuhn-Tucker necessary conditions. 


Be DIRECT MINIMIZATION AND MAXIMIZATION BY GRADIENT 
PROJECTION 


1. Gradient Projection Solution of a Sequence of Constrained 
Optimization Problems 


Since the necessary conditions in Rosen's algorithm are equi- 
valent to the Kuhn-Tucker conditions (see Appendix A), this method may 
be described in terms of two optimization problems, An arbitrary y, say 
y, , was chosen and the minimization problem 


min, 6x OS see (16) 


Subject to 


was solved for x,. Then a solution y, was found for the maximization 


problem 
SEY, cre 7) (17) 
Subject to 


Ay) 20 


This procedure was continued until some stopping criterion was met. A 


convenient stopping criterion is 
-] 
Litxi,vi) - f&y-3,vi-,) 1" < € 
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If this criterion is met, the point (x, , y,) found is the desired solution 
by the definition of a saddle point - - equation (4). There remain to be 
shown the conditions under which the sequence of points (x, ,X2,...) 
and (yo..¥1,-.-) converge to x* and y*. The Principle of Contraction 
Mapping [13] was applied to the class of functions considered in this 
thesis and conditions were found which ensure convergence for these 
functions (see Appendix B). 
Z. Simultaneous Minimaximization by Gradient Projection 

Before describing this method it should be stated that it does 
not seem to work in general; it is included primarily to point out some of 
the pitfalls of using two dimensions to visualize n-dimensional concepts. 

The problem of finding a saddle point of f(x,y) subject to 


aA, &) 20 and AW) 2 0 was altered in the following manner. Let 


i aes 
Ze |. ? r (z) = = 
7 a dX (y) 
= ~y 
Define the "gradient" of f(z) as 
-7 f£{(x,y) 
- nF zp =5- 1 
alg) TiGey) | = 


It was then intended to solve the Maximum Problem 


max , 12) 
Subject to 

Ray ee 

see Ne vv 


with the "gradient" as defined in equation (18), for 


18 


zk = oe (19) 


For the case where»x and y are scalars, the progress Otte algoritinimemay 
be followed on a contour plot and converges quite rapidly. Unfortunately 
this method does not seem to extend to n-dimensions except for isolated 


examples. 


Ls. 


IV. TWO-DIMENSIONAL EXAMPLES 
The methods described in the preceding section were initially 


tested on the following two-dimensional example. Find the saddle 


point of 
f(x,y) =et = vy"? + xy + 2x 4 4y = 6 (20) 
subject to 
1\V/ ea ae Ve 
x x 


(21) 
M- < y <M 
y y 


The function f is convex in X and concave in y, hence, by Theorems l 
and 2 a saddle point exists. The region of x,y space for which the 
inequalities (21) are satisfied is called the feasible region. If this 


feasible region is large enough, the saddle point will be interior and 


located at 
x4 — 16 
Vee lead 


Clearly, the selection of M.. M M. and M,, in (21) determines 
whether the solution will be interior or constrained. 
From equations (5) through (12) necessary conditions for (x*,y*) 


to be a saddle point are 


20 


= —2 —~ y* = = 
2Xx* -2 alias ee ee 0 
* 7 ' = 
SoS M.) Key 0 
east x i = 
(-x +M)) Lo 0 
(22) 
= x = * - a 
2y* -4 + x use Bee 0 
~ + aie = lh 
(y Hay 
-y*¥ +M_ )h = 0 
(-y y Hoo 
* oy te > 
Sat MoM. 0 Hy 0 
—-x* + M* = 0 I 2 0 
x . x2 (23) 
*+M 20 2 0 
y rl 
—v*¥ +M 286 ) 2 0 
i y ry2 


A, SOLUTION OF NONLINEAR SIMULTANEOUS EQUATIONS 
All three methods converged for the several cases of the previous 
example which were examined. 


With M = M\ = M.. a M = an = 3., the three methods 


converged from the starting point (1,-1,1,1,1,1) to 


x* = -1,6000 
y* = 1,2000 
78500 orale Adal emai Ou 


The Newton-Raphson method required four iterations, Brown's algorithm 
took three iterations, and the gradient projection root-finding method 
required five iterations. These results are shown in Figs. 1, 2, and 3 
respectively. 


veal 





Contours of f(x,y) = x* - y® + xy + 2x + 4y - 6 showing 


progress of the Newton-Raphson method to an interior saddle 


point, 


22 





riguee 


2 


Contours of f(x,y) = x* - y* + xy + 2x + 4y - 6 showing progress 


Brown's Algorithm to an interior saddle point. 


ZS 





iG a0 


Contours of f(x,y) = x*- y® + xy + 2x + 4y - 6 showing progress of 


gradient projection root-finding method to an interior saddle point. 
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M,, was changed to 1.0 and all three methods converged to the 


constrained saddle point 


xc y= — 1-5 000 
y* = 1,0000 
Dea? lesen? le - UO 
u = 0.59000 
y2 


in five, four, and six iterations respectively. These results are shown 
in Figs. 4, 5, and 6. Note that a nonfeasible starting point was used 
for the gradient ee root-finding method in Fig. 6. Also note 
that the method does not seem to make any progress from iteration 1 to 
iteration 2. This is due re the fact that only two of the six variables are 
Smown in Fig. 6. 
The objective function was then changed slightly (for reasons to 
be explained in a following section) to 
f(x,y) = x* - y® + Sxy + 2x + 4y - 6 
With M = 3., all three methods converged to the interior saddle-point 
Ne a 0) 8 SiS) 
ye = SO eo 


Di DIRECT MINIMIZATION AND MAXIMIZATION BY GRADIENT 
PROJECTION 


1. Gradient Projection Solution of a Sequence of Constrained 
Constrained Optimization Problems 
The function 
f(x,y) = x* - y* + xy + 2x + 4y - 6 (24) 
Vis 





ve 
Orel OUD 1.000 
0.730 0.797 
Cae 4 08 Degas 
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Fig. 4 
Contours of f(x,y) = x* - y* + xy + 2x + 4y - 6 showing progress of 


the Newton-Raphson method to a constrained saddle point. 


Z6 
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Contours of f(x,y) = x® - y® + xy + 2x + 4y - 6 showing progress of 


Brown's Algorithm to a constrained saddle point. 


ais 
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Pidmenc 
Contours of f(x,y) = 5 y° + xy + 2x + 4y - 6 showing progress of 


gradient projection root-finding method to a constrained saddle point. 
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was again used. A starting value of y was selected as 3, So the first 


stage of the method was to minimize 


f(x) = x* + 5x - 3 (25) 
subject to 

-3 <x <3 
A solution of x, = -2.5 was obtained. 


Then, the maximization problem 


max, f(y) = -y* + 1.5y - 4.75 (26) 


Subject to 

-3 Ss ys 3, 
was solved and the solution yg = 0.75 was obtained. This procedure was 
continued until the stopping criterion was less than 107°. The progress 
of the algorithm is shown in Fig. 7. The sequence of points (x,, 
Pe. acess s) ANG WY,, Vor..s.+++0s) ODViOUSlY Converges to xX* ama, = 

Again, the objective function was change to 

f(x,y) = x* - y* + Sxy + 2x + 4y - 6 : (27) 
This time the sequence of points (x,,y,) (starting from the origin) diverged 
as shown in Fig. 8. A sufficient condition for convergence is that the 
equation 

a ae, (28) 


be a contraction mapping (see Appendix B), where 


. F —_ 574 0 
a1 “Ty 7% 25/4 


Neither of the sufficient conditions for C to be a contraction mapping are 
Satisfied since 


ae 





Pg 7 


Contours of f(x,y) = x® - y® + xy + 2x + 4y - 6 showing progress of 


sequential min-max by gradient projection to an interior saddle point. 
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2 


Contours of f(x,y) = x* - y® +.5xy + 2x + 4y - 6 showing progress of 


sequential min-max by gradient projection. 
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Qigee ~ 
A ra l 6 > I . (3 0) 


Furthermore, a sufficient condition for C not to be a contraction mapping 


isesatisticd mile ees, 


625 
oi ‘ 
N 16 
paper ’ 
Note that these conditions are independent of the initial values_of x and 
Y- _—— 
_ ee LOSS aP Tx Lee © ES I UCM Le ALS acim 9 termes 


eS. r© to bea contraction mapping are satisfied, since 


1, thé conditions f 


ener 


2 = ee). “Se 
= | tae 


and 


2. Simultaneous Minimaximization by Gradient Projection 


~ Again, the function 


#9 | ero (x cy) n= x = y* + xy + 2x + 4y - 6 


SS ys 3S, 


is used to illustrated the method, 


The starting point x, = 1, y, = | was selected, and the nega- 


tive gradient with respect to x and the positive gradient with respect to y 


22 


were computed. These two components form the "plus-minus SINS ISI 
defined in equation (18). They are shown by heavy lines in Fig. 9. The 
true gradient and it's components are shown by dotted lines. Astepis 
taken in this "plus-minus" gradient direction. The gradient projection 
algorithm takes the maximum step in the direction of the gradient (in this 
case the plus-minus gradient), i.e., it moves as far as possible without 
leaving the bounded search region. In this case the point ie (-1.6,3.), 
which lies on the constraint boundary, is reached. The algorithm then 
determines whether or not this maximum step is too far by computing the 
inner product of the normalized projected gradient at point 0 and the 
gradient at point 1’. If this inner product is greater than or equal to zero 
the maximum step is taken. If it is negative, repeated linear interpolation 
is used to determine the point where the inner product is zero, i.e., 
where the gradient is orthogonal to the normalized projected gradient 
evaluated at the previous point. In this case, the maximum step is too 
far and by repeated linear interpolation the point 1 is obtained. This 


procedure was continued until the necessary conditions for convergence 


of the algorithm were met. 
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Figued 


Contours of f(x,y) = x* - y* + xy + 2x + 4y - 6 showing progress of 


simultaneous min-max by gradient projection to an interior saddle point. 
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V. ASPECIAL CASE 


In his investigation of two-sided allocation problems, Bartley {1 ] 
proposed a solution for a special type of saddle-point problem. 


A functional of the form 
(32) 


—0 

I = | K dt 
a 

is considered where K = K(x, y, t) is a real valued function. It is desired 


to find a function x which minimizes (32) subject to the constraints 


0 and , (33) 


ad 
> 0 and (34) 


i.e., a saddle point of the functional I is sought subject to the constraints 


(33) and (34). 
It is convenient to introduce a new functional 
ite ie ot 
J = [eee j @+ux+ wy) dt | (35) 


are constants (Lagrange multiplier's) to be found. Note 


where H and be 


that if x and y are admissible (equations (33) and (34) are satisfied) then 


J differs from I by the amount H. + ve 


Now if the functions x° and y° satisfy 
(36) 


Nore) icone) 


es a) = 
But they may not be admissible functions. 


they are a saddle-point of J. 


If they are admissible and satisfy (36) they are called the min-max 


PUategies, <~ and y*. 
80 


Bartley proposed that sa and - be selected arbitrarily. Then the 


Kuhn-Tucker necessary conditions for a saddle-point [14] can be applied 


to the integrand, H, of (35). These conditions are 


SMe, yoo, (OH (i 7 y Jax =0 , x 2.0 
OX Ox | 
(37) 
OulCcwmey es) =O. 4 (creek) y 4) y =mObm yor m0 
ay ay | 


for all te lagb we can be usedeto obtain a trial solutieme(x° ~"y°). If thie 
trial solution is admissible, then the min-max strategies have been found. 
If the trial solution is not admissible, Bartley used a specially tailored 
Newton-Raphson iteration to correct the value of (X, Y) and obtain a 
sequence approaching (1,1). 


A particular functional considered by Bartley was, 
cd 5 

J, y) = | Cex +2t)” + xyt - fy + 3t)* J dt (38) 
‘o) 


By dividing the interval [0,1] into N subintervals each of length T and 
assuming piecewise-constant x and y over T, the integrals can be 
approximated by summations. (Note the if the problem is to be solved on 
the computer there is a diScretization performed eventually anyway). 


Now the problem is in the form: Find a saddle point of 


A k k k 
fj. =TS Ch) +2) + xyk)—— - Ve) + 3) eo 
D N N N 

k=0 

Subject to 
x (k) 2 0 a Ii = Oe, N=! 
(40) 

Vik) 270 = 0 se Ne 
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and 


a 
Ma 
x 
ae 
I 
—_— 


(41) 


ri 
M2 
‘< 
ue 
HH 
— 


where the saddle point of Jy which is a function of 2N variables subject 
to 2N+4 constraints (if the equalities are written as two inequalities as 
in equation (15) ), can be found by the methods described previously. 

This example was solved by the sequential min-max gradient 
projection method with N=2Z5 and gave the trajectories shown in Fig. 10. 
These correspond quite closely to Bartley's results, also shown in Fig. 
nO. 

The simultaneous min-max method was also tried on this problem 
and gave the same results as the sequential method. Although, as 
mentioned previously, the simultaneous min-max method does not seem 
to extend in general to examples of dimension greater than second-order, 
the feasible region in this case was apparently so restricted by the 


equalities that the algorithm was able to converge. 


e/) 





Sequential Min-Max by 
Gradient Projection 


Bartley's results 





Fig. k0 


Functions x and y which are a saddle point of 


p= [ [&+2y?4 xyt - (y + 3t)?] dt 


subject to |i xdt=1, x 20, jtydt=1,y20. 
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VI. A PURSUIT-EVASION PROBLEM 
A pursuit-evasion problem was also considered in which each 
player's dynamic system is described by linear differential equations and 
the payoff is a weighted combination of terminal miss distance and each 
player's control effort expended. 
The state equations of two systems are 
=) 3 A (tx (igi a Bu, fe ality) oer 


(t) 


Pas 
a= e 


i! 
DS 
= 
|x 
@ 
~~ 
he) 
+ 
ow 
@ 
~~, 
ee 
le 
o) 
=S 
Ne 
= 
ee 
l 


~o\tq? ~ Seo (42) 


where x and x are n-vectors describing the respective players' states 
disamy time t, a and u, are q-dimensional control vectors ior cach 


player ,’ and Et is q-dimensional Euclidean space. The problem is to 


find a saddle point u*. u.*) of the payoff 


T= Kt) -x tJ)” Q(x tt) - xt) 
(43) 
ie TE T 
+] flaw - ut) - u(t) Rw, (t)] dt 


where Q is a positive semidefinite matrix and a and R are positive 


definite matrices. This is essentially the problem solved by Ho, Bryson, 
and Baron [6]. The solution is well known and has been presented by 


many authors, including Hutchison and Permenter [7], and Willman [20]. 


’ The dimension of each player's control vector may be different 


although for simplicity they are assumed equal here. 


3g 


Additional constraints on the control variables are assumed, such 
as 


lu .(t)| < M_ , hee eC (44) 
lu .(t)|< M_, eee ere (45) 


Any other equality or inequality constraints on either the state or control 
variables may also be included (Jacob and Polak [10] include energy 
constraints), but for simplicity only the above are considered. The cal- 
culus of variations as an analytical tool is no longer as attractive as it 
was in the unconstrained problem. Although Berkowitz, in his definitive 
paper based on the calculus of variations [3] , considered constraints 
theoretically, apparently variational techniques have not been applied 
to solve meaningful examples with constraints. This deficiency is 
important because it is quite realistic to expect constraints on the con- 
trol variables of physical systems, e.g., aircraft do not have unlimited 
control surface deflection nor unlimited acceleration. Furthermore, ina 
pursuit-evasion situation it is realistic to expect control variables to 
always be at their limits, e.g., an aircraft commander will use every 
bit of maneuverability built into his aircraft to avoid a pursuing missile. 

The procedure used to solve this problem was to assume that the 
control variables are piecewise constant, and to A/S OSAP ENTE the differ- 
ential equations by difference equations and the integral in the payoff 
functional by a summation. Then the nonlinear programming techniques 
discussed previously were applied. 
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If the control variables are piecewise constant, the state equations 


can be written as 


(k) 


x (k+l) = aM (k+1)T, kT)x_ Ak) + ta (k+1)TkT)u on 


x (k+1) = ( (k+1)T,kT)x_ (k) + Be (k+1)T,kT)u. (k) 


where 27 (k+1)T,kT) and € (k+1)T,kT) are the respective state 


transition matrices [12], and TI (k+1)T kn), ro (k+1)T, kT) are 


given by 
(k+1)T 
P( (k+1)T,kq) “| €_( (k+1)T,t)B (t) dt (47) 
(k+1)T 
P ( (k41)T, kq) aa &_( (k+1)T,t)Bi(t) dt (48) 


respectively. Although the following development can be performed for 
time-varying linear systems, for notational simplicity it is assumed that 


Att), A. (tale. BA, and Be (t) are constant matrices in which case 


q (k+1)T,kT) = € (7) 
Ga (Ki be € (7) 
r (k+1)T,kT) = Di) 
rt Uae EE) D7) 


_ ‘i 
Is ce Lx (N)-x, (N)] Q [x (N) - x (N)] 
(49) 
N-1l ~ - 
7 1) i Lu_(k) Ss u (k) - u (k) iss u. (k) ] 


4] 


Writing out a few terms for the pursuer's system gives 


i (T) . Yu (0) 
aie ) © (T)x (0) + P (Tu, (0) 
ee, = 2 x) ‘ Pfu) 
= \ : | \ 
® L& (Tx, (0) + Tu, (0) + Pu @) 


= a f 
= PT)x_, (0) + en ies P Mua) 


u (0 
~p 


N=] eZ 


— N \ 
x (N) ts (T)x,, (0) + € CTE Tu (0) * yy CET ae) 


+...4 2 (7) Pu (N-2) + Pu (N-1)  . 60) 


A similar expression can be written for 2 (N). Thus it is possible to 
express the final state values in terms of the initial state and the N 
values of the control vector. Substituting equation (50) into equation 


(49) allows the payoff to be written as a function of the 2Nq control 


values 
2 set 
tl) u_ (1) 
u = | P , uF a (51) 
~p ica —& a 
u(N-1) u. (N~1) 
The nxNq matrices ¢ and q_ are defined as 
os N-1l | IN 2 
a = 1 q Ah Saco eo 
@,= [8 @I m) &@rio: TM] ga) 
= N-1l ee = _—s 
cn [ee (T)T A(T) | oo (T)T «T) | TS) | (53) 


42 


the NQxNq matrices Rand oe are given by 


R 0 
p 
0 R ee 
p 
boh= ; ; ; 94 
p : : (54) 
0 0 R 
R 0 0 
© 
0 R 0 
mn e 
R = (55) 
= 
0 0 R 
= 
and 
~ a N 
9 = bees ¢ (T)x_ (0) (S6) 


where @ is the "predicted terminal miss" at time t=NT if no controls are 
applied during the preceding N intervals. 


Jy can now be written as 


- I ee z- 4 2 
5 = LO+¢ wu UL Qlere u. éu.J+Th, R, 


a aero b - <+T.- 4- -T <-T ilk 
+2fu ( Qel-u [@ Q€ Ju -u [Qe]} +6 Qe. 67) 
a0) p p p e-e “ee 
Since it was assumed that Q is positive semidefinite and Ss positive 
definite, it follows that the matrix [é° 8 . + Ro is positive definite 


amemence that Jy leva COmie Une om ot u, ki Jy is also a concave 


NAC TIOn O , then Jy will have a saddle point. This will be the case 


el 
=e 
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= 


ae : ; Ta 
if the matrix Ro is weighted relative to ¢ ads in such a way that 


Qe. = RJ is negative semidefinite. 


i Af one = 
Assuming that [@ oo - RJ is negative semidefinite, the pre- 


viously described methods can be applied. The constraints (44) and (45) 


are written as 


I 
= q _ 
aoG,) = |---| a, - Cm) #9 (58) 
q 
I 
Z a 
doG) = fa] a, - Gm) 29 (59) 
q 


where Ly is the G-—dimensional identity matrix, m_ is the 2q-dimensit@mem 
p 

VecCclorO! COMolamis oak m. is the 2q-dimensional vector of constants 

M.: and the problem may be stated in the concise form: Determine a 


saddle point of (57) subject to the constraints (58) and (59). 


For u*,u) to be a solution to this problem, it is: necessary and 


iS Ubnalenteolewinlavete (u* us) and the 2 q-dimensional vectors Be and BL. 


satisfy 
7- u*,u*) + -] = (0 
a,b“ we) Ug ; aes = 
= 7 ll 

A Wat = 0 

~ p oL Eo 
» (u* > 0 
dU) 0 
2 Q 
By U 
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ie 1D) 
== 
- \T 
* — 
Aus) w 0 
» (u*) ae 
e = 


The following simplified pursuit-evasion game was solved by using 
a Newton-Raphson method, Brown's algorithm, and by sequential minimi- 


zation and maximization using the gradient projection algorithm. 


Let 
0 1 
A=A (t) =A (t) = 
pi= AL g je 
0 
B=B (t) = B (t) = 
a(t) = B,(t) 
i! 0 
Q = 
0 0 
r 0 (r 0 
p e 
R = : ia = 
P 0 ¢ 0 r 
p e 
0 LO 
x (0) = x (0) = 
Te 2 ie -2 


The initial time was set at 0, the final time at 2, and the sample interva! 
was taken as T=l. 


The state transition matrix is 


\|—_— 
tr 
i) 
-_— 
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and the matrix [ is given by 
i A ie = 


Jy becomes 


_T O74 374 7 0 7 
_ ; Ye 
3/4 1/4 0 in 
(61) 
-3 9/4 3/4 -3 
si v4 fe = mn ue oe + a 
P = ae 3/4 174 ae | 
and for In to have a saddle point it is sufficient that 
13 > 0 
p 
= 
r 9/4 
Note that Jy is aeeenvex function of uy for c > - = but since Re was 


assumed positive definite it is necessary that = See 
Values of i lene os 9. were initially chosen and with Bs = 
M = 2,., the Newton-Raphson method, Brown's algorithm and sequential 


min-max with gradient projection gave the interior saddle-point 


II 


03.200 


u, (0) 1.000 u_ (0) 


ee ee : 


c 
~_ 
— 
— 
Il 


Oe hers 


The Newton-Raphson method and Brown's algorithm both converged in 
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four iterations while the gradient projection method required 24 minimiza- 
tions and maximizations. 
The value of as was then changed to ue = =0.8. In six iterations, 


the sequential min-max method converged to 


uw, (0) ="07800 u, (0) Ou3:2.0 


ui) UMSos u, (1) Oman 


The Newton-Raphson method required four iterations to converge to the 
same point. Depending on the starting point, Brown's algorithm converged 


to the unconstrained solution or created a singularity in the vicinity of the 


constrained solution, but it did not converge to the constrained solution. 
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VII. CONCLUSIONS 

Nonlinear programming methods have been applied to the solution 
of constrained saddle~point problems, in particular, to a type of problem 
arising in purSuit-evasion games with inequality constraints on the values 
of the control variables. The payoff functions considered were real- 
valued, concave-convex guadratic polynomials, although nonlinear 
programming methods are not, in general, restricted to such functions. 

Five techniques were suggested = - the two most promising being 
the Newton-Raphson method to solve nonlinear simultaneous equations 
obtained from the Kuhn-Tucker necessary conditions, and the solution of 
a sequence of minimization-maximization problems using the gradient 
projection algorithm. The Newton-Raphson method was found to be some- 
what sensitive to the starting point chosen. Gradient projection, while 
insensitive to the starting point (even a nonfeasible point is acceptable), 
was found to converge only when certain conditions on the function were 
satisfied. These conditions were found by an application of the Contrac- 
tion Mapping Principle. 

The primary advantage of these methods is that they admit inequality 
constraints on the variables, in fact, it is required that the variables be 
constrained. 

A serious disadvantage (at least in the case of pursuit-evasion 
problems) is that only an open-loop strategy is obtained. A pursuit- 


evasion problem with inequality constraints for which a closed-loop, or 
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feedback, strategy is desired must (apparently) be approached using 
calculus of variations techniques [4]. 

In addition to the inclusion of inequality constraints, differential 
game theory has many unanswered questions. Perhaps the most obvious 
flaw in the formulation of pursuit~evasion games is the lack of realism in 
the performance criterion. It has been pointed out [7] that an aircraft 
commander being pursued by a missile will be unsatisfied with a strategy 
which tells him to conserve his energy, yet it was found that for a saddle- 
point to exist the evader's control energy term must be weighted rather 
heavily in comparison to the terminal-miss term. This raises a very 
fundamental question concerning what is meant by "optimal" in optimal 
strategies. In the formulation given in this thesis, optimal means mini-~ 
max strategies which are a saddle point of the performance criterion. 
But, just as in optimal control theory, the engineer or analyst must be 
able to formulate realistic performance criteria if the "optimal" control, 
or strategy, is to have physical meaning. 

A more realistic payoff for pursuit-evasion games would be simply 
a measure of the terminal miss, but a saddle point may not exist for this 
criterion and the meaning of the term "Solution" must be altered. Very 
little information is available on differential games for which a saddle | 
point does not exist. It is well known that matrix games that do not have 
pure strategy solutions (no saddle point exists) have solutions in mixed 
strategies. A strategy is called "mixed" if it is chosen from a set of 


admissible strategies in accordance with some particular probability 
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distribution. Berkowitz [4] discusses the notion of mixed strategies 
for differential games but observes that the mathematical difficulties 
which arise have not been overcome. 

Another approach which warrants investigation for possible 
application to differential game theory is introduced by Salmon [18] . 
Although limiting his attention to the design of controllers for systems 
with uncertain parameters, Salmon proposes an algorithm to solve a 
class of minimax problems for which a saddle point need not exist. 

Further investigation of the methods suggested in this thesis 
should include the solution of meaningful examples, e.g., a pursuit- 
evasion problem with enough samples over the time interval to adequately 
represent the trajectories. It is expected that the conditions for con- 
vergence of the sequential min-max method would prove somewhat 
restrictive, Difficulty with convergence of Brown's algorithm has already 
been experienced and could be expected with the Newton-Raphson method 


in larger dimension systems. 
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APPENDIX A 
In this appendix it is shown that Theorem 4 of Rosen's gradient 
projection algorithm [15] is equivalent to the Kuhn-Tucker necessary 
conditions for the nonlinear programming problem. 
Kuhn and Tucker stated the Maximum Problem as 
max f(x) ' x an m-ve@ier 
subject toA(x) 20 , X a k-vector 
x 20 
The necessary conditions for x* to be a solution to this problem 


are"that x* and "sSome p satisfy: 


k 
Fe) = ye + Fy de <a (Al) 
‘ i 
{7,6 +5 My, yeedIp xr = 0 (A2) 

x* 2 0 (A3) 
A(x*) 2 0 (A4) 
A)" w= 0 (AS) 
2-2 (A6) 


where F(x) is the Lagrangian function, F(x)=f(x) + u d(x) and the con- 
straints X(x)2 0, x20 must satisfy the constraint qualification[14] . 
Rosen [15 ] considered only linear constraints in Part I of his algorithm, 


and linear constraints always satisfy the constraint qualification. 


ol 


The linear constraints can always be written in the form 


N@)=xn, -b 2 0, i=l,...k (A7) 


m 
> (1.9*s =a Se ee : (A8) 
ij 
jae 

Note that 
‘B , &) = n, : (AQ) 


The mxk matrix N, is defined as 

ee ee (Al10) 
(for linear constraints this corresponds to F° in (14]), and the k-vector 
bgais 

Mae ice) aa | (All) 


Using these definitions, the system of k inequalities can be written as 


T 
Ny x*- b = Q (Al 2) 
In addition Rosen does not require that x 2 0Q; therefore, (Al) must 
hold as an equality, and the necessary conditions become 


vot) + Nie pee 0 


Ne x* - bz 0 
(A13) 


II 
© 


(Ne x*- bl]? p 
ts ek 
Rosen has assumed that the k constraints form a convex, bounded 
region R. Thus, it must be that k2m+l. Clearly, at most m of the k 
hyperplanes A(x*) = 0 are linearly independent. Let ne, es aa are era © (7 


a2 


be a set of linearly independent unit vectors which define a set of q 


linearly independent hyperplanes 


d , &*) = 0 »wl=l, 2p. l= q Sen . (Al 4) 
i d , &*) > 0 foralli, i=l,...,k@, then 2 must cquar 0. Wiamtnis 


Case X* is an interior polnt of the region R and the necessary Conaimions 


reduce to 


ee SS = 0 : (A15) 


the familiar necessary condition for an unconstrained problem. 
Now assuming that d , &*) = 0, i=l,... ,qG, which can be the case 
for at most m of the k hyperplanes, the mxq matrix 


Na le, (Al 6) 


can be formed from the vectors defining the above hyperplanes. Also, 
letting Lo be the vector of the H. which correspond to d , &*) =, iors 


Kuhn-Tucker necessary conditions can be written as 


f(x* = 
12 Cid ts NG me 0 (Al7) 
A, &*) = 0 , i=l,....4 (Al 8) 
ay = 0 (A19) 


q 


Using the psuedo-inverse of Ny to find BY from (Al7) gives 


ee ee a ee " 
Bee Ne eae eee : (A20) 


From (Al9) this must be 20 , or 


(Ne N,) Na [7 fa) ] so. (A2 1) 


aye. 


Equation (Al7) can now be written as 


T — | eel 
7 {(x*) + N_(N N N —-7 {£(x* = 0 
fet) + Ny(N, Ny) NS L-7,f@4) 1= 0 
Al -] _T 
I - N_(N_ N N 7 to) = 0 B22 
L m . 5s : ] Fe 0 (A22) 
Using Rosen's definition of the projection matrix 
P = I-N (NN ane (A23) 
q Gq Gq q fe | 
equation (A22) becomes 
* = 
Pa Gas ] 0 (A24) 


Equations (A21) and (A24) are precisely the necessary conditions 
given by Rosen [15] in Theorem 4 for x* to be a constrained maximum of 
f(x). If £(x) is a concave function of x the Kuhn-Tucker conditions, and 
thus equations (A21) and (A24), are also sufficient for x* to be a con- 


strained global maximum of f(x). 
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APPENDIX B 

The Contraction Mapping Principle [13] can be applied to find 
sufficient conditions for a sequence of points obtained from the sequential 
minimization-maximization of a real-valued, concave-convex function 
with continuous and bounded second partial derivatives to converge to 
the saddle point. 

A mapping A of the space R into itself is said to be a contraction 
if there exists a number a@< 1 such that 

p (Ax, Ay) < ap (&.y) 
for any two points x,y €R, where Ris an arbitrary metric space, and p 


genotes the cheleeioisihe distance in R. In this thesisegems demncdsa- 
n 1 4 
S Ak il 
py ={% &, -v)? }* ={txvl” bev fF . 
i=l " 


The functions examined in this thesis may all be represented by 


Quadratic polynomials of the form 


f(x,y) =X AX +y By +xXc+xXDy+tyetg . (B1) 


This can be reduced by linear transformations to a simpler form by intro- 


ducing new vectors x and y defined as 


[x 


x= xX +h 


_ (B2) 
Ve Val 


where h and k are constant vectors to be determined. Substituting (B2) 


into (Bl) gives 
f(x,y) = (th) A(kth) + (y+k)’ B(ytk) + (x+th)'c 


+ (xth)” D(y+k) + (ytk) e +9 (B3) 
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which can be written as 
I TL ah alk 
fOomy) —ex mex ye By + X Dy ex (ZAh + c + Dk) 


+ y (2Bk + e + Dh) 
+(h AL +k Bkt+hoct+h De+k'e+g) . (84) 


T 
By SClLUMG tle cocmlelentsuanr x? and y equal to zero, the values of h 


andeks.are.determined from 


Zl Fak + Cc = 0 

Dhware choker ¢ = 0 
OG, 

Piece) ula ce | 


D 281k y le (BS) 


If the matrix « A has an inverse, (B5) can be solved for h and k. 
D Pe 


This matrix will have an inverse if Ais positive definite and B is negative 
definite. Equation (B4) then has the form 


Be) = x 


where Ais positive definite, B is negative definite, K is a constant, and 


Ax + y By + x Dy + K (B6) 


f has a saddle point at the origin. 
Allowing the variables x and y to be unconstrained, for a particular 
y the function f is minimized when Vf, y) =Q. The value of x at this 


point is 


tI 
lo 


Vix, y) = 2Ax + Dy 


tI 
roll 
> 
A 
we 
K< 


es (B7) 
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Similarly, for any x, the funcuousisicemasaini 2Zeduat 


guar) = ee eae 


7 
at 


(B8) 


Met x2, ,y¥, be an arbitrary point. Minimizing =? with especie x, the 


next value of x (x, ) will be 


Xx, = “HA™ Dy, (B9) 
Maximizing f(x,,y) with respect to y gives as the new value of y 
oa) aa -* B* Dx, 
See me Dye Dy, R (B10) 
Minimizing f(«,y,,,) with respect to x obtains 
X44. 7 -3 At Dy ; +1 
= = Appt IDRC (Bl 1) 
Equations (B10) and (Bll) can be written 
Zi41 = OZy (B12) 
where 
, pa 
24 ve 
and 
= AT DBs Des 0 
c SS as eee ee a ee ee 
0 | 4B°DA*D 
If Z,;,, is closer to the origin (the saddle point) than z, , i.e., if 
218 
(ae | ~ 1 z,| where | 24 | = { : (Z4, } , the sequential min- 


max procedure will converge. This will be the case if (B12) is a 
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contraction mapping. From page 45 [13], a sufficient condition for the 


mapping C to be a contraction is 


Doe Ly, a Pa = (mel (B13) 
where the a, , are elements of C. 

An alternative result may be derived from the definition of a con- 
traction mapping. A mapping 

Z= CZ 
WS a contraction if 

o (Cz) <a pz) (B14) 


where a< 1 , thatis, 


zC Cz < a zz (B15) 
Ths, it 
zicicz <-ziz (B16) 


then the mapping C is a contraction. Conversely if 


T 
zC C2 ZZ (B17) 


then © is net a contraction mapping. 


It is well-known that on the sphere ||z|| =1, the maximum value of 


z'c'cz aS 


max 2'C Cz = di (B18) 


At 
where i, is the largest eigenvalue of C C, and that the minimum value 
is 


min ple Cz =z (B19) 
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where SN is the smallest eigenvalue of cic. The matrix clc is real, 


symmetric and positive definite, hence its eigenvalues are all real and 
positive. 


From (B18) and (B19) 
Mcsalidall® <0? CeCe <wtee  n (820) 


Looking at the right-hand inequality first 


zccz < d, llzll’ , (B21) 


ifexg = 1 thén it follows that 


zccz < zz = |zll (B22) 
eneethe mapping’ C is a contraction. 
From the left-hand inequality 
Ayllzil s gz e°cz (823) 
>. 1 
and if Ay ; 
zoCz > zz (B24) 


ena © is not a contraction mapping. 
Thus;)A, — 1. is a suiiieient condition ter Grte be alcenivaction 
mapping and A > lis a sufficient condition for C not to be a contraction 


mapping. 
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